Slow initialization

[holocronweaver]

Plots.jl had very slow initialization for me on Julia 0.5.1, usually taking ~10-20 seconds to complete the initial using Plots. Then the call pyplot() would entail another 5-10 seconds. The problem seems slightly worse on Julia 0.6 rc3. I am using a Samsung SSD on a decently beefy machine.

Has anyone else been experiencing this?

In case it helps, I attached a profile created in Julia 0.6 rc3.

Profile.init(delay=0.1)
@profile using Plots
Profile.print()

And here is timing:

@time using Plots
...dozen or so deprecation warnings...
 12.528709 seconds (5.46 M allocations: 308.239 MiB, 2.39% gc time)

[mkborregaard]

Unfortunately, this is the current situation. Part of it is that precompilation has been deactivated on release versions of Plots at the request of the METADATA maintainers.
It also seems much worse for me on 0.6 rc3 than it did on 0.6 rc2.

[mkborregaard]

The Profile shows that every single second is spent in inference.jl

[holocronweaver]

Thanks for the quick response!
I am guessing this means this may be a Julia bug which needs to be reported?

[mkborregaard]

I'll see if I have the old 0.6-2 binary lying around to do a comparison

[holocronweaver]

I compiled 0.6 rc2 and re-ran the timing and profiling. Basically the same as 0.6 rc3. I can check 0.5.1 as well if you like, but I am guessing it will turn up the same.

@time using Plots
 12.522696 seconds (5.74 M allocations: 324.781 MiB, 2.43% gc time)

[mkborregaard]

Oh, thanks! I guess this means no regression on rc3. We're discussing the more general Plots issue here: #918

[mauro3]

X-ref: #357

[krvajal]

Is there any update on this. Pretty sad that this is the package recommended for plotting in the Julia docs, a language that advertises itself as been fast, and plotting a single data point takes more than 12 seconds, just loading the plotting package itself

[pkofod]

Is there any update on this. Pretty sad that this is the package recommended for plotting in the Julia docs, a language that advertises itself as been fast, and plotting a single data point takes more than 12 seconds, just loading the plotting package itself

Plots.jl has an overarching philosophy that is very different from "time to first plot is paramount". My guess is that you can load any of the backends on their own, and it will be much faster. If it's worth it in lost productivity due to awkward syntax is totally up to you. Plots is features first, time to plot Nth.

I hope I don't come off as harsh, but believe me, this is a very tricky thing to get right as proven in the hundreds if not thousands of comments debating this topic. Comments made by smart people, and people who are very well-versed Julia programmers.

Plots.jl was made to make the actual plotting a pleasant experience. We all want to decrease load times, improve how conditional loading is done, and so on and so forth, but the magic solution is not here yet.

Progress is being made in this and other packages, but remember where the effort comes from in a project like Plots.jl. Combine a tricky problem with limited supply of work hours (though some people still put in a lot of hours mind you), and you know why this is not yet resolved, if it can even ever be fully resolved in Plots.jl.

The people who use and enjoy Plots.jl are probably those who enjoy how easy it is to use. People who are endlessly annoyed with load times probably use or develop other packages. Plenty out there for you to try, or you could even just call Matplotlib or ggplot2 from Julia using PyCall or RCall.

[holocronweaver]

So....no update then?

[pkofod]

[David-OConnor]

I think the confusion comes from this package being listed at the top of Julia's official plotting page. Someone new to Julia who's expecting a fast language tailored to numerical and math/science use might be dissuaded by each plot taking 30s to 1 min to make. Eg the commands "using Plots", "pyplot()", and "plot(...)" at the top of this official page each take 10-20s on my system.

[cosama]

@David-OConnor That exactly happened to me. I installed Julia tried some simple plotting task, waited like 10 times for the plot to not show up for 20-30 seconds and went back to using python. My workflow is based on executing the same script many times, adding a few new things each time. I think the long start up time can be a huge deterrent, particular to people that are not aware if the slowness being related mainly to the plotting library and not julia in general. It would be at least good to add a warning to https://julialang.org/downloads/plotting.html.

[mkborregaard]

So, it is very much a Julia issue in general. The problem is that Plots is written in Julia, and thus needs to precompile the first time you run any plotting command. Other plotting packages like pyplot, GR and plotlyjs are written in other languages (python, C and javascript) with a thin Julia wrapper, so they don't suffer from this problem.

It also means that there's no "quick fix" - we can't suddenly magically change something to avoid this slow initialization, which is honestly one of Julia's biggest issues. It is also very much on the radar of the core devs (it is key priority number 3 listed here: https://discourse.julialang.org/t/compiler-work-priorities/17623 ) and much work is also going into building a new interpreter which could alleviate some of this: JuliaDebug/ASTInterpreter2.jl#37 . In addition, Julia now supports static compilation of packages which alleviates this problem: https://discourse.julialang.org/t/ann-packagecompiler-with-incremental-system-images/20489 which works with Plots: https://nextjournal.com/sdanisch/plots-remix . One thing we do know for sure, though, is that complaining in an issue here on Plots is unlikely to improve the situation.

But the more general issue is also one of workflow. Most Python users are accustomed to a workflow where you write a script, then run that script from bash. Then change a comma, and run the script again. I believe that people develop the aesthetic that it's somehow a "clean" way of working with code. You can do that with Julia, but it's not a very good workflow exactly because it requires recompilation of all your methods every single time you run the code. Putting the body of your script in a function instead, and then calling that function from an open julia session, suffers from 0 of these problems, leads to immediate updates and plots after the first plot in a session and is every bit as clean.

When you go to a new language, you just might consider using the native workflow of that language, which is to work with functions and modules in an active Julia session (possibly with Revise.jl). Or, if you insist on using the Python workflow, maybe not be surprised when you encounter the limitations of that workflow?

[mkborregaard]

I suggest that anyone considering commenting further on this issue make sure they have read the previous comments.

[cosama]

My intent was not to complain. I just wanted to make sure that other people in a similar situation will not just move to some other system but actually get the information why this is happening and how they can mitigate this. I opened an issue to add some information to the julia web page.

I think you added some good links here and pointed people in the right direction by mentioning that calling code from a open julia session is the way to go (using include to run scripts seems the simplest to me). I agree that different languages require different workflows. However, it is crucial to point this out to new users. The parts of the documentations I looked into did not provide me with this information. I would also like to add another link here on how to work with Jupyter and that there is a plug in for atom, which also help to work around the long loading time.

[Bebotron]

Should this also be affecting editing Jupyter cells? I understand that running a cell to plot something may take a while, but should it also be affecting the sluggishness of editing the cell? I don't know if this is related to this Julia problem or if its a Jupyter problem. But basically when I try to edit the cell that will generate my plots it is very laggy. Has anyone else experienced this, or can anyone direct me to a better source if that's too unrelated to this thread?

[pkofod]

Should this also be affecting editing Jupyter cells? I understand that running a cell to plot something may take a while, but should it also be affecting the sluggishness of editing the cell? I don't know if this is related to this Julia problem or if its a Jupyter problem. But basically when I try to edit the cell that will generate my plots it is very laggy. Has anyone else experienced this, or can anyone direct me to a better source if that's too unrelated to this thread?

I think it would be helpful if you were more specific, but really I think you should ask on discourse. Are you plotting 30 million scatter points, or just drawing a line for example? But really, ask on discourse.julialang.org :)

[Bebotron]

It ended up being a problem with printing plots in svg format, and later a problem with GR in Plots, which was fixed by simply updating my packages #2111. Sorry for the vagueness and digression!