Doubt in dataloader_next_batch()

[bijouvj]

I am wondering why the current_position is advanced by (BT) * sizeof(int) instead of (BT + 1) * sizeof(int) here:

  // read the B*T+1 integers from the file into batch
   fseek(loader->tokens_file, loader->current_position, SEEK_SET);
   fread(loader->batch, sizeof(int), B*T+1, loader->tokens_file);
   // advance the current position by B*T integers
   loader->current_position += B*T * sizeof(int); <<== HERE

[karpathy]

the inputs stride by B*T.
the targets are just offset by 1 from inputs.

[PeterZhizhin]

Didn't notice, opened a bug: #321

I agree that the current position should probably be also advanced by B * T + 1. Otherwise on the next batch, the first token we read is the last target token.

Also, with current implementation, reading 2 batches of size N is not the same as reading a single batch of size N*2. Stylistically, I think they should be the same.