Skip to content
This repository has been archived by the owner on Sep 1, 2022. It is now read-only.

Performance Issue with high variable and dimension count #1295

Open
GTOL opened this issue Jul 3, 2019 · 0 comments
Open

Performance Issue with high variable and dimension count #1295

GTOL opened this issue Jul 3, 2019 · 0 comments

Comments

@GTOL
Copy link

GTOL commented Jul 3, 2019

I try to use CDF format to store my data so that each piece of data is using one dimension and two variables. I noticed that the loading speed of the CDF file is increasing significantly with the dimension and variable count.
I also run a simple test on this issue. The dimension length is randomized from 1 to 10000 for each data, and the data size is from 1000 to 10000 with a step of 1000 (for example, a data size of 1000 will give dimension count of 1000 and variable count of 2000).
Here is the result:
Size: 1000 Time: 48 ms
Size: 2000 Time: 133 ms
Size: 3000 Time: 150 ms
Size: 4000 Time: 404 ms
Size: 5000 Time: 622 ms
Size: 6000 Time: 1426 ms
Size: 7000 Time: 2374 ms
Size: 8000 Time: 2773 ms
Size: 9000 Time: 4044 ms
Size: 10000 Time: 5159 ms
Maybe you can find a way to fix this? Thank you.

The file format is netcdf3.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant