About ReceivedPacketsDropped (once again)

[davekc]

Hello @lathoub,

First, thank you for your valuable work with this library!

I tried it on a Teensy 4.1, but I lose packets here and there (more on Windows with rtpMIDI v1.1.14 than on macOS, but it still happens and even chokes the session and disconnects). Obviously more with a lot of MIDI messages and less when there are fewer. Surely has to do with this wiki on Windows and that discussion on macOS (but the mentioned branch is unfortunately no longer available to try)...

#define USE_QN_ETHERNET false

#if USE_QN_ETHERNET
  #include <QNEthernet.h> // lwIP
  using namespace qindesign::network;
#else
  #include <NativeEthernet.h>
  const byte mac[] = { 0xDE, 0xAD, 0xBE, 0xEF, 0xFE, 0xED };
#endif

// AppleMIDI (aka rtpMIDI)
#define ONE_PARTICIPANT
#define NO_SESSION_NAME
#define USE_EXT_CALLBACKS
#include <AppleMIDI.h>

APPLEMIDI_CREATE_DEFAULTSESSION_INSTANCE();

void setup() {
#if USE_QN_ETHERNET
  Ethernet.begin();
  if (!Ethernet.waitForLocalIP(5000)) {
#else
  Ethernet.setSocketSize(8192); // 2048 by default
  if (!Ethernet.begin(mac)) {
#endif
    Serial.println(F("Failed DHCP!"));
    for (;;);
  }

  Serial.print(Ethernet.localIP());
  Serial.printf(":%d\n", AppleMIDI.getPort());

  MIDI.begin(MIDI_CHANNEL_OMNI);

  AppleMIDI.setHandleConnected([](const APPLEMIDI_NAMESPACE::ssrc_t& ssrc, const char* name) {
    Serial.println(F("Connected"));
  });
  
  AppleMIDI.setHandleDisconnected([](const APPLEMIDI_NAMESPACE::ssrc_t& ssrc) {
    Serial.println(F("Disconnected"));
  });

#ifdef USE_EXT_CALLBACKS
  AppleMIDI.setHandleException([](const APPLEMIDI_NAMESPACE::ssrc_t& ssrc, const APPLEMIDI_NAMESPACE::Exception& e, const int32_t value) {
    switch (e) {
      case APPLEMIDI_NAMESPACE::Exception::BufferFullException:
        Serial.println(F("*** BufferFullException"));
        break;
      case APPLEMIDI_NAMESPACE::Exception::ParseException:
        Serial.println(F("*** ParseException"));
        break;
      case APPLEMIDI_NAMESPACE::Exception::TooManyParticipantsException:
        Serial.println(F("*** TooManyParticipantsException"));
        break;
      case APPLEMIDI_NAMESPACE::Exception::UnexpectedInviteException:
        Serial.println(F("*** UnexpectedInviteException"));
        break;
      case APPLEMIDI_NAMESPACE::Exception::ParticipantNotFoundException:
        Serial.println(F("*** ParticipantNotFoundException"));
        break;
      case APPLEMIDI_NAMESPACE::Exception::ComputerNotInDirectory:
        Serial.println(F("*** ComputerNotInDirectory"));
        break;
      case APPLEMIDI_NAMESPACE::Exception::NotAcceptingAnyone:
        Serial.println(F("*** NotAcceptingAnyone"));
        break;
      case APPLEMIDI_NAMESPACE::Exception::ListenerTimeOutException:
        Serial.println(F("*** ListenerTimeOutException"));
        break;
      case APPLEMIDI_NAMESPACE::Exception::MaxAttemptsException:
        Serial.println(F("*** MaxAttemptsException"));
        break;
      case APPLEMIDI_NAMESPACE::Exception::NoResponseFromConnectionRequestException:
        Serial.println(F("*** NoResponseFromConnectionRequestException"));
        break;
      case APPLEMIDI_NAMESPACE::Exception::SendPacketsDropped:
        Serial.println(F("*** SendPacketsDropped"));
        break;
      case APPLEMIDI_NAMESPACE::Exception::ReceivedPacketsDropped:
        Serial.println(F("*** ReceivedPacketsDropped"));
        break;
      case APPLEMIDI_NAMESPACE::Exception::UdpBeginPacketFailed:
        Serial.println(F("*** UdpBeginPacketFailed"));
        break;
    }
  });
#endif
}

void loop() {
  MIDI.read();
}

From what I remember, I tried:

• Playing with NativeEthernet socket size setting (as you can see)
• Playing a lot with QNEthernet settings (including PBUF_POOL_SIZE, IP_REASS_MAX_PBUFS, DEFAULT_UDP_RECVMBOX_SIZE, MEMP_NUM_REASSDATA, LWIP_SO_RCVBUF, LWIP_PBUF_REF_T, etc.)
• Playing with your AppleMIDI settings (UdpTxPacketMaxSize and MaxBufferSize)
• Enabling the rtpMIDI diagnostic log... nothing!
• Using the latest code in the Master branch
• Reading all the closed issues and all available discussions
• Trying this:

while (readDataPackets()) parseDataPackets();
while (readControlPackets()) parseControlPackets();

Anyway, is there something else I can try? Because otherwise, I wonder if this communication protocol is even reliable or not. Maybe the library could still be optimized, but I have the impression that it will always be impossible to be perfectly reliable (even on a local network it’s still UDP packets that can get lost, right?). It’s beyond my skills, though.

Thank you in advance for your feedback,
Dave

[lathoub]

Do you have a list of the incoming MIDI commands? (a Wireshark dump helps!)

[davekc]

I will do that as soon as possible. I have installed Wireshark but still never used it. I checked manually, when I had a ReceivedPacketsDropped exception, and it was clearly missing a MIDI message. But I understand you want to see the whole thing...

[lathoub]

But I understand you want to see the whole thing...

Correct, without a dump from Wireshark, i can't do anything
(in the Wireshark filter field, enter applemidi || rtpmidi)

[davekc]

I've tried but Wireshark is too greedy on my CPU to work on my desktop. I will try tomorrow with a 24-core server I have.

In the meantime, I found this:

RTP-MIDI (IETF RFC 6295) is a specification for sending/receiving standard "MIDI 1.0" messages using standard networking protocols [...]. RTP-MIDI includes a data recovery mechanism (MIDI event journaling) to address packet loss that can occur on networks, eliminating the need for packet retransmission (which would increase latency and reduce throughput).

What is it exactly? Is it implemented in the library?

[lathoub]

No, journaling is not implemented: the way journaling is setup does not match the resources (memory and compute capabilities) available on a microcontroller. Rtpmidi is over-engineered, but works ghese days as udp packages are not dropped anymore on a local network.

i’m surprised to read wireshark is to greedy on resources on your machine - i run it on a 10 year old MBAir without an issue.

[davekc]

First, is this the kind of output you want? Need something more?

I only captured one missing event, so you can tell me whether this is what you want or not. The green lines are the source, the white ones are looped back MIDI messages from the MCU. So the selected line is missing and triggered a ReceivedPacketsDropped exception. I can grab a few more to spot any possible patterns.

Then, are the journaling would be possible to implement on Teensy 4.x? Kind of powerfull MCU. If so, would you be open to add this mechanism to the library? I would be willing to pay for this. I really need to be 100% sure that nothing is missing. But maybe my problem is somewhere else too...

Finally, I was surprised that you were surprised about Wireshark. So I tried again (with v4.0) and it worked well.

Again, thanks for your help!

[lathoub]

I'm not a fan of implementing journaling, as it does not fit the microcontroller platform (even if Teensies are giants in a small footprint).

Not sure why you drop commands, I suggest you simplify your setup (hardware and software) to the bare minimum. Who/what generates the NoteOn/Off (and at what rate) and how do you get them to your Teensy (what is the network infrastructure)? Simplify that first and then build it up again until you see dropped packets. (I'm not ruling out flaws in the code, just trying to identify where drops happen).

Alternatively, you can add debug messages in the code to print incoming packets and see where/how they get dropped (See DEBUG statements in the code)

Arduino-AppleMIDI-Library/src/rtpMIDI_Parser.h

Lines 30 to 45 in d8a4e98

	size_t _bytesToFlush = 0;
	protected:
	void debugPrintBuffer(RtpBuffer_t &buffer)
	{
	#ifdef DEBUG
	for (int i = 0; i < buffer.size(); i++)
	{
	SerialMon.print(" ");
	SerialMon.print(i);
	SerialMon.print(i < 10 ? " " : " ");
	}
	for (int i = 0; i < buffer.size(); i++)
	{
	SerialMon.print("0x");
	SerialMon.print(buffer[i] < 16 ? "0" : "");

[davekc]

A very interesting reading, thank you for sharing it with me. Which leads to the conclusion of the question I asked myself from the start: is rtpMIDI only reliable? The answer, obviously now, is no. Most of the time, but I cannot rely on "most of the time". So, reluctantly, I am going to move towards another approach, sadly. And, you are right, the journaling is not the miracle solution I was hoping. Moreover, iConnectivy has not implemented it on their products either (they have confirmed this to me).

I had already reduced to a bare minimum. Only a small piece of code (shared from the beginning with you) on a Teensy 4.1. Nothing else to compute. Plugged into a 1Gbps switch with a 6-inch cable. Computer plugged right next to it, just doing that... playing a MIDI file over rtpMIDI. It still happens. The only thing is perhaps the speed of the events. Even if it’s not always 200BPM music, I need to find something 100% reliable. Even a single dropped packet would be too much. I can’t afford to lose a single NoteOn event...

That said, I’m going to test the DEBUG feature, we never know. Thanks for pointing that out.

Many thanks for your generosity, your answers, your time and your efforts to help people and to develop this library!

[lathoub]

Can you share some code (send code (on what?) and receive code for Teensy 4.1).

I did modify the code recently and you make me doubt my modifs. Maybe i can reproduce the issue

[davekc]

You make me doubt myself... let me redo a few more tests, under ideal conditions, to come up with a reproducible procedure. That will be next week, though.