r/StableDiffusion
•
u/arjan_M
•
Apr 17 '23
•
1
Discussion I mad a python script the lets you scribble with SD in realtime
5
4
u/Majestic-Class-2459 Apr 17 '23
This is awesome.
I developed an extension for A1111, let me know if you needed any help.
4
u/arjan_M Apr 17 '23
Thanks for offering. It is only a simple script I made.
It could use some extra little features, but I don't know if it is the worth the trouble to put too much effort in it, because I would have to make a complete paint program.
It would be better to implement something like this in photoshop or Krita.
I am surprised something like this isn't already made besides that photoshop script
997
u/nxde_ai Apr 17 '23
in realtime
My GPU say: nope, not gonna happen
64
u/Chalupa_89 Apr 17 '23
My GPU: "Real time? is there any time that isn't real? Time is relative anyway."
11
Apr 17 '23
I mean, if you have money to throw at it, Google Colab is always an option.
→ More replies (3)4
u/RMCPhoto Apr 17 '23
About how much does it cost you to run on colab?
12
Apr 17 '23
It's $9.99 for 100 compute hours. You can probably get 200 512x512 @ 20 samples in one compute hour. Of course adding samples, resolution increases the VRAM usage.
I mean, it's an absolute steal for what it is. For $9.99 (and through your Google account) you can run any ipynb juniper notebook file (essentially a combo bash + markdown + python setup) and you have access to a 16GB GPU.
Here's the one I run: https://gist.github.com/jonshipman/b17a3a66a0c2247dc001f84fa671b434 I just updated it for ControlNet1.1.
And, for a point of reference, I'm 38 years old. I've spent $60 in roughly a month. Admitting I could have been more conservative about my usage - I have forgotten and left it on for a bit and I've played around with various video extensions. Only mention the age bit as I have a bit more messing around money than a 20yo.
You do get a few hours gratis when you first use it. AND you do get unlimited CPU usage for free. I've heard you can modify the script to use the CPU, but that would mean hours for one image (and colab does kick you off by way of a recaptcha every 2 hours or so).
You can absolutely get away with spending less per month. If I was smarter about it I could make $30 last a month, just by fucking around less - but where's the fun in that? It's also been way funner than MidJourney. MidJourney was ok, but you need to have an idea in your head first. With colab + sd, you only really need the beginning of an idea. Like "man on a beach" then batch out 10 and figure out step 2. Like "man on beach, throwing frisbee", etc, etc. Being able to customize the batch size is nice and let me burn through the $$$ with colab.
I'm only using MJ in comparison as I know they are $10 per month. Colab has a monthly plan, but AFAIK, that's only for command line access and faster computers - you'd still need to buy compute units ala carte. So really, it comes down to how much you use it. If you were wild with it (100 steps, 1024x1024) you could probably burn through $10 in 4 days. I know starting out I burned through it pretty quickly - and video burns through it quickly depending on FPS.
Is it worth it? Luckily you should be able to test 5 compute units (give or take) if you've never used it before. Copy my gist, upload it to colab, and click run. It does need you to add your own models (civitai is a good source for those) as I didn't need TheLastBen's self downloading stuff.
→ More replies (3)→ More replies (80)5
2
u/Arcuis Apr 17 '23
That does look cool though. I'd love to use something like that.
6
u/arjan_M Apr 17 '23
I put in on github so if you want to try it.
houseofsecrets/SdPaint: Stable Diffusion Painting (github.com)
You do need atomatic1111 with controlnet installed and a capable graphics card.→ More replies (2)
2
u/buckjohnston Apr 18 '23
Very cool, where are the images stored, just in ram? If I wanted to keep one I would just to alt and print screen button correct?
Edit: So far the only issue I'm having is it's not totally following my scribbles and sort of just sort of making random images with custom models I'm using.
2
u/arjan_M Apr 18 '23
You can try different controlnet models. in the json file you change the name of the model you want. You need to download them and place them in the right folder yourself tough.
I also made an update that lets you save the image with the s key→ More replies (1)
78
u/7734128 Apr 17 '23
It would be super frustrating if the "perfect image" flash past on the left only for you to accidentally scribble a tiny dot after that and lose the picture forever.
37
u/poly_lama Apr 17 '23
Open a GitHub issue for local caching. Extremely simple problem to solve, you could even open a PR to do it yourself if you ask ChatGPT how (assuming you don't know python)
0
5
u/GunslingerBara Apr 17 '23
Could add a ctrl+z option to undo the last change, or some other command for pixel-by-pixel reversals.
107
u/Atovange Apr 17 '23
Looks awesome, perfect example of how AI can help artists and not bring them down
47
u/bubblebooy Apr 17 '23
Also an artist won’t take that output and use that as the final piece. They will use their skills touching up and adding to it until it matches their vision and quality level.
22
u/Put_It_All_On_Blck Apr 17 '23
Today and in the near future, yes.
But not in a few years. Look at the progress we've made recently, hands for example were a big issue, and you'd always have to fix them or paint them and let the AI take a few dozen attempts at fixing them. Now some models are able to do hands as good as the rest of the image.
There will always need to be some added input, as an AI isn't going to decipher your sentence and scribble into EXACTLY what you were thinking. But you don't need an artist per say to make a good AI image, you just need someone that understands art.
→ More replies (3)13
u/Jonas_Wepeel Apr 17 '23
I mean, look just a little further than that. Soon these systems will be self prompting and fully generating entertainment with no human involvement. We’ll be watching season 34 of Friends and people will say “you would never know it’s completely AI generated.” Hopefully man made media and art continues and becomes ‘artisanal’ lol. Assuming we all live to see that day.
→ More replies (3)→ More replies (1)3
u/Wildercard Apr 17 '23
The artist will probably need not a single-layer picture, but an AI enhanced picture with all the layers they can then adjust to their will manually.
8
u/FreightCrater Apr 17 '23 edited Apr 17 '23
I derive no enjoyment from having the skills I've practiced for 10 years automated. Would a dancer be happy just pressing a button instead of moving their body? Would a race driver be content with a self driving car?
Financially, sure. There'll now be no other way to stay competitive but to use ai generated images. But I have no interest in continuing as a professional artist if it means relying on ai generation, and I'm not alone.
People already don't want to pay professional artists anything more than peanuts, and now we'll be outcompeted by millions of talentless prompt monkeys.
There's no putting the genie back in the bottle, but yeah, I'm salty.
8
u/Atovange Apr 17 '23
People used to be payed to make portraits, then photography came by and... some people are still payed to make oil portraits. You could also reduce photography just to point and shoot but it's of course more than that. I think AI art is a bit like photography in that sense, though different.
→ More replies (5)8
u/Divinum_Fulmen Apr 17 '23
This is dumb. A painting can put someone "in their best light," you can't do that with a photo, unless you know, paint over it (which is still what Photoshop is doing).
Also, many paintings, scratch that, nearly all paintings are of things that don't exist, or didn't exist at the time of the painting. Photography never came close to replacing art. This is just the lie that deniers of the change that's happening keep telling themselves.
Humans are being replaced, and not just in art.
→ More replies (7)→ More replies (32)5
u/TashaSkyUp Apr 18 '23
The sad part for me is that your ability to be a truly free artist was stolen by capitalism. Think about it, would you be so salty about this if capitalism never forced you to into working so hard to learn the "in demand" version of your skills? Maybe, you would have taken up other mediums or discovered that actually writing or something else was your thing. But no you where likeley like most of forced to specialize, to optimize so that you can make enough money to not die.
6
u/FreightCrater Apr 18 '23
I would not be as salty, I think you're right. That's a really valid insight into how I feel about this.
→ More replies (1)10
u/Tommy_Boy97 Apr 17 '23
By doing all the work for them..?
Seems more like the human is helping the AI "create" art, than it is the other way around.15
u/Atovange Apr 17 '23
You could say that, the human is doing the composition and the AI is choosing colors and adding details. Sounds like a good way to learn art. Like you start just by the composition, then you also sketch the colors and the AI fills the rest.
→ More replies (43)→ More replies (11)7
u/hawara160421 Apr 17 '23
True-ish but op can actually draw. I challenge the average SD-artist to get simple things like the hat, the feet and proportions as well as the general composition this right with a few strokes, lol. Right side actually, genuinely, looks like a Mobius panel in black and white.
I mean, you could just draw awkward stick figures and hope it positions them in a non-awkward way but then you could as well just type in a prompt.
6
Apr 17 '23
It's the same discussion as it was with photography, digital photography, computer graphics, computer-based image edition, point-and-shoot cameras, phone cameras, etc...
Art is dead is a common knee-jerk for the last 200 years.
→ More replies (27)5
u/MinosAristos Apr 17 '23
The difference with AI is that it's a much broader tool than e.g cameras which can produce only a very specific kind of art. Photographs definitely did hurt hyperreal painting, because the results are hard to discern. Likewise the advancements in digital design and printing hurt the art of manual media creation.
I think AI will in the coming decades make the creation of all sorts of art far easier and make many or almost all current art creation methods obsolete.
People with art skills are likely to still be best able to work with AI tools to make the best art, but the skill floor will be much lower.
→ More replies (14)4
u/xDraGooN966 Apr 17 '23
I hate that AI "discussion". Only people up in their feelings on both sides spouting such extreme crap about AI with 100% certainty.
Anyone that says AI will take literally all jobs and no human will have a single job is stupid.
Anyone that says AI is so stupid it can't do anything at all and will not replace anyone is stupid.
When the truth probably lies somewhere in the middle.
AI will improve work flow for a lot of people doing a multitude of jobs, so fewer people will be needed to get the same work done. That however still means over time there will be more and more people out of jobs.
Which won't matter anyway because in 50-100 years in a +3.5°C - 5°C world, we will be too busy fighting for water and food and survival anyway regardless of jobs.
→ More replies (1)
63
17
23
u/sindivila Apr 17 '23
Guys someone else already posted this script few days ago. You can find it here: https://openai.art/study/unleashing-creative-power-of-stable-diffusion-showcase/
14
u/sishgupta Apr 17 '23
Little bit different but the same idea
4
u/sindivila Apr 17 '23
yeah, I agree. I commented before OP shared the repo and I'm glad he did. For some reason, I thought that he was dangling a carrot in front of us and I didn't want people to feel disappointed in case the OP just disappeared all of a sudden so I shared that code
7
6
6
u/DoingItWrongly Apr 17 '23
Cries in AMD gpu
6
u/Selto_Black Apr 17 '23
... there has been an amd fork for a1111 for awhile now though.
https://github.com/AUTOMATIC1111/stable-diffusion-webui/wiki/Install-and-Run-on-AMD-GPUs
→ More replies (1)
15
u/Warm_Celery3890 Apr 17 '23
That’s the stuff I joined Reddit for. Great! Please share the script ☺️
5
7
u/karterbr Apr 17 '23
!RemindMe 3 days
3
u/RemindMeBot Apr 17 '23 edited Apr 18 '23
I will be messaging you in 3 days on 2023-04-20 12:12:15 UTC to remind you of this link
21 OTHERS CLICKED THIS LINK to send a PM to also be reminded and to reduce spam.
Parent commenter can delete this message to hide from others.
Info Custom Your Reminders Feedback
1
u/dogtierstatus Apr 17 '23
GitHub link?
2
u/arjan_M Apr 17 '23
houseofsecrets/SdPaint: Stable Diffusion Painting (github.com)
Let me know if it works for you
1
u/sasquish Apr 17 '23 edited Apr 17 '23
EDIT:
I managed to get it up, however it does not change when more strokes are added, error mention something about model missing
2
u/arjan_M Apr 17 '23
Make sure you have the right model. It should have the exact same as the name in the payload.json file. You also copy paste the name of the controlnet model from the webui to the payload.json In the extra folder is a text file with the names of controlnet models
1
u/elreyfalcon Apr 17 '23
I’m confused as to why you are promoting and scribbling simultaneously? Does it add more keys to your prompt or what? source code would be helpful
2
1
u/afacanx Apr 18 '23
That is increadible. Is it possible to learn this power? Or just use it :)
2
u/arjan_M Apr 18 '23
You can download it here:
houseofsecrets/SdPaint: Stable Diffusion Painting (github.com)
You do need automatic1111 webui with the controlnet extension installed on your computer though
1
u/ASharkThatEatsPizza Apr 18 '23 edited Apr 18 '23
Hey so I got everything installed and running but it doesn't update the image after each stroke. I'm getting a fresh image if I save a new seed or whatever in the payload.json then draw any random line, but whatever I draw doesn't seem to make any difference to the actual image generated. It seems like it's only taking the prompt set in the payload to generate an image no different than if I were simply using the txt2img prompt. Any idea what I might be missing here to prevent me from live updating as I draw?
Also I submitted a pull request for a linux variant of your start script
1
1
u/Longjumping-Fall-150 Apr 19 '23
Dumb question but how to install this? Github says "Run the Start.bat file" and that's it. Nothing is happening on my end. I launch A1111, then run Start.bat. A command line window opens, downloads some stuff and then closes. Is this supposed to be run in A1111 SD folder somewhere?
2
u/arjan_M Apr 19 '23
It shouldnt close after you run the start.bat. It is probably an error. Try running it in the cmd. So go to the sdpaint folder and in the location bar in type in cmd. The in the command window type python Scripts/SdPaint.py And reply here what error you get
→ More replies (1)
3
3
3
3
u/ilinamorato Apr 17 '23
This is absolutely the way some artists are going to work in the future. Then they'll take the output and paint details that the AI missed. Beautiful work.
3
u/Corrupttothethrones Apr 17 '23
Works great. I just had to update the payload for the new inline module and scribble model. Ill look at modifying the script tomorrow to incorporate the prompt into the app, dont like having to switch to the payload.json.
3
u/derdigga Apr 17 '23
Reminds me of fry from Futurama, where he traded his hands with robo devil to be able to play/paint amazingly. What I'm saying is with bad input he could archive what he had in his mind.it's kinda amazing where we are now in time, enabling this to everybody.
3
u/Mr___Medic Apr 17 '23
I'm seeing something like this for the first time and to say I'm impressed would be a huge understatement. It makes me want to buy an actual PC more than video games, and I've always liked the latter.
3
u/sloppies Apr 17 '23
I try to be a pessimist because I have been disappointed by news headlines for the last decade straight, but between this stuff and ChatGPT, I am so shocked at how quickly AI is moving right now.
2
2
2
2
2
Apr 17 '23
Does anything you're using to create this (automatic1111 api, stable diffusion, etc) require a paid subscription to use, or can this be done for free? If it's paid, can you estimate how much?
I'm interested in setting this up, but am starting from square 1 and would need a pretty handheld walkthrough of what to do. Please let me know if you can give me and others like me a hand :)
→ More replies (1)
2
u/danamir_ Apr 17 '23
Thanks a lot, I was looking for a script like this !
I made a fork and a few improvements for my taste, like support for other resolutions and HR fix. See the issue #7 on your repo.
See ya,
2
2
2
2
2
2
2
u/Boycandy Apr 22 '23
There is a lot of potential here. Imagine having this as a dedicated app, together with cloud computing! Would be perfect for tablets (e.g. iPad Pro) Untapped market. Opportunity of a lifetime!
2
2
1
1
1
u/cobra_economics Apr 17 '23
Please make the script available, preferably as a plugin for automstic1111. Good work!
1
u/Polyglot-Onigiri Apr 17 '23
That’s pretty cool. I never thought of trying to make something like this with Python. Can’t wait to see the script
1
1
u/Corrupttothethrones Apr 17 '23
Seems like an adaptation of the post 3 days ago, they used photoshop and it updated every time they hit save.
1
u/andyzzone Apr 17 '23
+1 Vote Up
what i was actually thinking about since ai was introduced...how to install it to SD?
1
1
1
1
1
1
1
u/Reasonable_Barber484 Apr 17 '23
Nice script! Thanks for sharing! Do you think it's possible to connect ir to a drawing tablet instead of the mouse?
1
1
1
u/tehSlothman Apr 17 '23
I'm a little confused, what does this do that hitting generate forever and using controlnet scribble mode doesn't?
1
1
u/Mold-Mschool Apr 17 '23
Can you imagine this being in an interactive art museum or something? I mean, yes, teenagers and the sort will abuse it, but still it would be amazing. Especially as a stand alone product.
2
1
1
1
u/Maty9009 Apr 17 '23
Awesome! Although the AI itself probably won't even recognize stick figures of me....
1
1
u/intotheirishole Apr 17 '23
Your prompt engineering needs work.
You need to say 'Moebius!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!'
Also 'Trending in instagram'
.
/jk
1
1
1
1
1
1
u/Longjumping-Rub-3317 Apr 17 '23
Does anyone know how to do something like this and make it even closer to real time? I tried to generate smaller images and reduce the number of steps to 3-4. The result is very fast, but it is far from being RT. my gpu is a 3080 ti
→ More replies (1)
1
1
u/SwimForLiars Apr 17 '23
It would be cool if you could have an overlay mode where you draw on top of the picture instead of a parallel picture, like it's filling in the blanks. Like the one on the right applied as a transparent+black on top of the generated one, instead of being a white+black on the side.
I don't know if the pictures would match 1:1 though.
1
u/KangTheMighty Apr 17 '23
That's incredible! Is there a tutorial available where I could learn to set that up?
1
1
u/Zanzaben Apr 17 '23
This is amazing. The last time I tried to use SD I couldn't get it to make what I wanted and was thinking if only I could do a crappy sketch for it.
1
u/watchforwaspess Apr 17 '23
This is amazing. Will it work on an RTX 3060 12gb? I guess I’ll have to try.
1
1
1
1
1
u/apost8n8 Apr 17 '23
I'm so excited to see stuff like this. I want to make some amazing self guided AI produced creativity. What a neat tool!
1
u/B4-711 Apr 17 '23
I haven't had too much success with creating Moebius style pictures. Is this the standard 1.5 model?
1
1
Apr 17 '23
I dont know how coding works, but from my simpleton perspective, this is very impressive! Its an amazing tool you have created, imagine you vaguely remember a dream for example, you cant start drawing what you do remember and see if anything starts to feel familiar as it trials and errors. Amazing stuff
1
u/Famberlight Apr 17 '23
I'm getting this error when trying to scribble anything
In Start.bat
Exception in thread Thread-1 (send_request):
Traceback (most recent call last):
File "C:\Users\xxxxx\AppData\Local\Programs\Python\Python310\lib\threading.py", line 1016, in _bootstrap_inner
self.run()
File "C:\Users\xxxxx\AppData\Local\Programs\Python\Python310\lib\threading.py", line 953, in run
self._target(*self._args, **self._kwargs)
File "C:\Stable_Diffusion\stable-diffusion-webui\SdPaint\Scripts\SdPaint.py", line 92, in send_request
return_img = r['images'][0]
KeyError: 'images'
In webui-user.bat
API error: POST: http://127.0.0.1:7860/controlnet/txt2img {'error': 'IndexError', 'detail': '', 'body': '', 'errors': 'list index out of range'}
Traceback (most recent call last):
File "C:\Stable_Diffusion\stable-diffusion-webui\venv\lib\site-packages\anyio\streams\memory.py", line 94, in receive
return self.receive_nowait()
File "C:\Stable_Diffusion\stable-diffusion-webui\venv\lib\site-packages\anyio\streams\memory.py", line 89, in receive_nowait
raise WouldBlock
anyio.WouldBlock
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\Stable_Diffusion\stable-diffusion-webui\venv\lib\site-packages\starlette\middleware\base.py", line 78, in call_next
message = await recv_stream.receive()
File "C:\Stable_Diffusion\stable-diffusion-webui\venv\lib\site-packages\anyio\streams\memory.py", line 114, in receive
raise EndOfStream
anyio.EndOfStream
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\Stable_Diffusion\stable-diffusion-webui\modules\api\api.py", line 145, in exception_handling
return await call_next(request)
File "C:\Stable_Diffusion\stable-diffusion-webui\venv\lib\site-packages\starlette\middleware\base.py", line 84, in call_next
raise app_exc
File "C:\Stable_Diffusion\stable-diffusion-webui\venv\lib\site-packages\starlette\middleware\base.py", line 70, in coro
await self.app(scope, receive_or_disconnect, send_no_error)
....
File "C:\Stable_Diffusion\stable-diffusion-webui\venv\lib\site-packages\fastapi\routing.py", line 163, in run_endpoint_function
return await dependant.call(**values)
File "C:\Stable_Diffusion\stable-diffusion-webui\extensions\sd-webui-controlnet\scripts\api.py", line 154, in txt2img
cn_image = Image.open(io.BytesIO(base64.b64decode(controlnet_input_image[0])))
IndexError: list index out of range
I added --api, "Allow other script to control this extension" option is enabled and updated auto1111 (git pull). ControlNet works fine inside of Auto1111
3
1
u/--DemonsWrath-- Apr 17 '23
That's actually amazing for concept art. Is it something people can get their hands on? The effort you have gone into to make this is great!
1
1
1
1
u/GhostSniper7 Apr 17 '23
*me watching this post and video*
my integrated GPU to poor 1050:- Ay bro , he looking at some new ways to torture you.
*1050 sweats in the corner of the case*
1
u/psycholepzy Apr 17 '23
Does this bridge the gap between being a genuine creator and getting an AI art assistant? Instead of relying entirely on prompt engineering, artists can instill their creativity into the work while producing something specific to them much faster.
At least, it'll make hasty character art for D&D easier.
1
u/Put_It_All_On_Blck Apr 17 '23
This is the future of artists.
Their jobs will pivot from drawing everything from scratch to knowing how to use AI to create the image they want. Digital art was already a baby step in that direction, with people using existing fonts, brushes, filters, and using other images online to assemble the image.
1
1
1
1
1
1
1
1
1
u/BazilBup Apr 17 '23
Wow this is amazing. However I'll never own anything that could do that in real-time
1
1
u/tatankadave Apr 17 '23
This is cool, I can imagine using this for dnd to bring quick life to characters and places
1
1
1
1
1
1
u/chartporn Apr 17 '23
You should embed a gif the animation shown here into the readme of your github repo.
1
1
1
u/feelmedoyou Apr 17 '23
This is pretty much the future of all illustration software, a form of drawing/painting auto-complete. Nice to see it up and running now, and can’t wait for the fine-tuning and mass adoption.
1
1
1
u/DudeVisuals Apr 17 '23
Holy molly someone call adobe …. They will offer you lots of money and steal your idea … Adobe scribbleFusion…. But Maybe choose a better sounding name
1
u/AlmightyBracket Apr 17 '23
I would love to try this out but I have no idea how to get something like this running on my PC. Can I get some help?
1
u/Think_Description_84 Apr 17 '23
This is amazing and I would love to play with it. Unfortunately when it comes to setting up environments Im dumb as brick. Can someone give stupid simple instructions or an installer or something?
I'd be very grateful!
1
1
1
1
1
u/xPiNGx Apr 17 '23
Very cool, awesome work. Would really help if you could save the image otherwise not sure why anyone would use this.
1
u/Educational_Lychee70 Apr 17 '23
This is great. Let’s just add a refresh button or a X second refresh timer for the slower GPUs. I can see this being really useful.
1
1
u/variant-exhibition Apr 17 '23
What kind of style is that your model is reacting with on demand to your lineart?

543
u/arjan_M Apr 17 '23 edited Apr 17 '23
It is a simple python script that gives you a canvas where you can paint on the right side and I will update on the left side. In the background it sends an updated image after every paint stroke to the API of automatic1111 and it uses the controlnet scribble model to generate an image.The video is in 4x speed, but it updates quite fast. It really depends on the speed of your graphics card and the controlnet model.It also works with controlnet 1.1 and that is a little bit faster.I can put the script online if anyone is interested.
here is the github repo
houseofsecrets/SdPaint: Stable Diffusion Painting (github.com)