story to images

[zulihit]

Thank you for your wonderful work. May I ask if it is possible for me to input a story and output a series of images. How can I implement this function

[haoningwu3639]

Well, since different people input stories in various ways (direct input or calling GPT to generate), we did not implement a function/interface for non-researchers.
But you can divide the story into prompts one by one, and generate the story frames step by step in an auto-regressive way by using the functions in 'inference.py'.
You can also further encapsulate a loop based on it to auto-regressively generate the image sequence corresponding to the entire story at once.

[zulihit]

Well, since different people input stories in various ways (direct input or calling GPT to generate), we did not implement a function/interface for non-researchers. But you can divide the story into prompts one by one, and generate the story frames step by step in an auto-regressive way by using the functions in 'inference.py'. You can also further encapsulate a loop based on it to auto-regressively generate the image sequence corresponding to the entire story at once.

Thank you for your reply. I noticed in inference.py that there is the following code:

prompt = "The white cat is running after the black-haired man."
prev_p = ["The black-haired man", "The white cat."]
ref_image = ["./data/boy.jpg", "./data/whitecat1.jpg"]
I couldn't find the files boy.jpg and whitecat1.jpg in the data folder. Could you please provide a prompt list and ref_images related to Figure 1 in the paper or another simple story, as well as instructions on how to run it and show the results? I would be very grateful.

[zulihit]

Thank you for your reply. May I inquire about the correct operation of inferrence.py, as I noticed that the two reference images are not in the data folder.

…

________________________________ 发件人: Haoning Wu ***@***.***> 发送时间: 星期三, 七月 24, 2024 10:00:14 下午 收件人: haoningwu3639/StoryGen ***@***.***> 抄送: zuli ***@***.***>; Author ***@***.***> 主题: Re: [haoningwu3639/StoryGen] story to images (Issue #33) Well, since different people input stories in various ways (direct input or calling GPT to generate), we did not implement a function/interface for non-researchers. But you can divide the story into prompts one by one, and generate the story frames step by step in an auto-regressive way by using the functions in 'inference.py'. You can also further encapsulate a loop based on it to auto-regressively generate the image sequence corresponding to the entire story at once. ― Reply to this email directly, view it on GitHub<#33 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AQLSFPARQVU4WS6FUFXSWVLZN6XOVAVCNFSM6AAAAABLMKQZZ6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDENBYGA2DGOJZGI>. You are receiving this because you authored the thread.Message ID: ***@***.***>

[haoningwu3639]

Sorry for the late reply, I was on vacation last week.
You can consider using the simple mode first, that is, taking one frame as the context condition.
First, use the single-frame version of our model, that is, single-frame inference without context conditions to generate a story protagonist, such as "The black-haired man"; Then use our auto-regressive mode to use prev_p and ref_image as conditions to continue generating the next frame.