Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add support for RLOG R2 #83

Merged
merged 4 commits into from
Mar 13, 2024
Merged

Add support for RLOG R2 #83

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
ad212a9
Add support for RLOG R2
jwbonner Mar 12, 2024
5768b56
Merge branch 'main' into rlog-r2
jwbonner Mar 13, 2024
29d4230
Merge branch 'main' into rlog-r2
jwbonner Mar 13, 2024
264fd76
RLOG improvements
jwbonner Mar 13, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 12 additions & 7 deletions docs/RLOG-SPEC.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,13 +1,13 @@
# Byte Format for Robot Logs (.rlog)

> Note: AdvantageKit v2 (2023) replaced RLOG with WPILOG as its preferred logging format. RLOG continues to be supported by AdvantageScope to interact with older code.

## Log Revisions

The first byte represents the log format revision. The decoding device should always check whether it supports the specified revision before continuing. Below is a list of possible revisions:

- 0x00 = Invalid file. This is the first byte of logs produced before a revision number was included.
- 0x01 = Current revision. The log follows the specification described below.
- R1 = Supported by AdvantageKit v0.0.1-v1.8.1 and v2.2.0-v3.1.1. Uses a predefined set of field types.
- R2 = Supported by AdvantageKit v3.2.0 and newer. Uses string names for field types.

All values are stored in big endian order.

## Message Types

Expand All @@ -28,15 +28,20 @@ Each human-readable string key (e.g. "/DriveTrain/LeftPositionRadians") is repre
1. Key ID (short, 2 bytes)
2. Number of bytes in string (short, 2 bytes)
3. String key (UTF-8 encoded)
4. Number of bytes in string (short, 2 bytes) - Only if RLOG R2
5. String type (UTF-8 encoded) - Only if RLOG R2

## Field

Field messages represent a change to a single value. If a key is not provided for a given cycle, the robot code should set its type to "null". A field's type can change over the course of a log, though each cycle can only contain a single value for each key (e.g. a field cannot have both an integer and boolean value at the same time). The structure of these messages begins with the following information:
Field messages represent a change to a single value. The structure of these messages begins with the following information:

1. Key ID (short, 2 bytes)
2. Value type (1 byte)
2. Value type (1 byte) - Only if RLOG R1
2. Value length (short, 2 bytes) - Only if RLOG R2

For RLOG R2, the value can follow any format. By default, use the [WPILOG-specified data types](https://github.com/wpilibsuite/allwpilib/blob/main/wpiutil/doc/datalog.adoc#data-types).

The possible value types are listed below along with the format of the value. Null (0x00) does not include more information.
For RLOG R1, the possible value types are listed below along with the format of the value. Null (0x00) does not include more information.

### Boolean (0x01)

Expand Down
101 changes: 39 additions & 62 deletions junction/core/src/org/littletonrobotics/junction/rlog/RLOGEncoder.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,30 +23,25 @@
import org.littletonrobotics.junction.LogTable;
import org.littletonrobotics.junction.LogTable.LogValue;

/** Converts log tables to the RLOG format. */
/**
* Converts log tables to the RLOG format. Based on RLOG R2 with
* support for custom type strings.
*/
class RLOGEncoder {
public static final byte logRevision = (byte) 1;
public static final byte logRevision = (byte) 2;

private ByteBuffer nextOutput;
private boolean isFirstTable = true;
private LogTable lastTable = new LogTable(0);
private Map<String, Short> keyIDs = new HashMap<>();
private Map<String, String> keyTypes = new HashMap<>();
private short nextKeyID = 0;

/** Reads the encoded output of the last encoded table. */
public ByteBuffer getOutput() {
return nextOutput;
}

/**
* Encodes a single tables and returns the encoded output. Equivalent to calling
* "encodeTable()" and then "getOutput()"
*/
public ByteBuffer getOutput(LogTable table) {
encodeTable(table);
return nextOutput;
}

/**
* Returns data required to start a new receiver (full contents of last table +
* all key IDs).
Expand All @@ -62,7 +57,7 @@ public ByteBuffer getNewcomerData() {

// Encode key IDs
for (Map.Entry<String, Short> keyID : keyIDs.entrySet()) {
buffers.add(encodeKey(keyID.getValue(), keyID.getKey()));
buffers.add(encodeKey(keyID.getValue(), keyID.getKey(), keyTypes.get(keyID.getKey())));
}

// Encode fields
Expand All @@ -83,14 +78,14 @@ public ByteBuffer getNewcomerData() {
}

/** Encodes a single table and stores the result. */
public void encodeTable(LogTable table) {
public void encodeTable(LogTable table, boolean includeRevision) {
List<ByteBuffer> buffers = new ArrayList<>();

Map<String, LogValue> newMap = table.getAll(false);
Map<String, LogValue> oldMap = lastTable.getAll(false);

// Encode log revision
if (isFirstTable) {
if (isFirstTable && includeRevision) {
buffers.add(ByteBuffer.allocate(1).put(logRevision));
isFirstTable = false;
}
Expand All @@ -109,20 +104,13 @@ public void encodeTable(LogTable table) {
// Write new data
if (!keyIDs.containsKey(field.getKey())) {
keyIDs.put(field.getKey(), nextKeyID);
buffers.add(encodeKey(nextKeyID, field.getKey()));
keyTypes.put(field.getKey(), field.getValue().getWPILOGType());
buffers.add(encodeKey(nextKeyID, field.getKey(), field.getValue().getWPILOGType()));
nextKeyID++;
}
buffers.add(encodeValue(keyIDs.get(field.getKey()), newValue));
}

// Encode removed fields (no longer supported)
//
// for (Map.Entry<String, LogValue> field : oldMap.entrySet()) {
// if (!newMap.containsKey(field.getKey())) {
// buffers.add(encodeValue(keyIDs.get(field.getKey()), null));
// }
// }

// Update last table
lastTable = table;

Expand All @@ -144,14 +132,19 @@ private static ByteBuffer encodeTimestamp(double timestamp) {
return buffer;
}

private static ByteBuffer encodeKey(short keyID, String key) {
private static ByteBuffer encodeKey(short keyID, String key, String type) {
try {
byte[] keyBytes = key.getBytes("UTF-8");
ByteBuffer buffer = ByteBuffer.allocate(1 + Short.BYTES + Short.BYTES + keyBytes.length);
byte[] typeBytes = type.getBytes("UTF-8");
ByteBuffer buffer = ByteBuffer.allocate(
1 + Short.BYTES + Short.BYTES + keyBytes.length + Short.BYTES + typeBytes.length
);
buffer.put((byte) 1);
buffer.putShort(keyID);
buffer.putShort((short) keyBytes.length);
buffer.put(keyBytes);
buffer.putShort((short) typeBytes.length);
buffer.put(typeBytes);
return buffer;
} catch (UnsupportedEncodingException e) {
return ByteBuffer.allocate(0);
Expand All @@ -160,100 +153,84 @@ private static ByteBuffer encodeKey(short keyID, String key) {

private static ByteBuffer encodeValue(short keyID, LogValue value) {
try {
// Generate key and type buffer
ByteBuffer keyBuffer = ByteBuffer.allocate(1 + Short.BYTES + 1);
// Generate key and length buffer
ByteBuffer keyBuffer = ByteBuffer.allocate(1 + Short.BYTES + Short.BYTES);
keyBuffer.put((byte) 2);
keyBuffer.putShort(keyID);

// Generate value buffer (and type for key)
ByteBuffer valueBuffer;
switch (value.type) {
case Raw:
keyBuffer.put((byte) 10);
byte[] byteArray = value.getRaw();
valueBuffer = ByteBuffer.allocate(Short.BYTES + byteArray.length);
valueBuffer.putShort((short) byteArray.length);
valueBuffer = ByteBuffer.allocate(byteArray.length);
valueBuffer.put(byteArray);
break;
case Boolean:
keyBuffer.put((byte) 1);
valueBuffer = ByteBuffer.allocate(1).put(value.getBoolean() ? (byte) 1 : (byte) 0);
break;
case Integer: // Save as Integer (int32)
keyBuffer.put((byte) 3);
valueBuffer = ByteBuffer.allocate(Integer.BYTES).putInt((int) value.getInteger());
case Integer:
valueBuffer = ByteBuffer.allocate(Long.BYTES).putLong(value.getInteger());
break;
case Float: // Save as Double
keyBuffer.put((byte) 5);
valueBuffer = ByteBuffer.allocate(Double.BYTES).putDouble(value.getFloat());
case Float:
valueBuffer = ByteBuffer.allocate(Float.BYTES).putFloat(value.getFloat());
break;
case Double:
keyBuffer.put((byte) 5);
valueBuffer = ByteBuffer.allocate(Double.BYTES).putDouble(value.getDouble());
break;
case String:
keyBuffer.put((byte) 7);
String stringValue = value.getString();
byte[] stringBytes = stringValue.getBytes("UTF-8");
valueBuffer = ByteBuffer.allocate(Short.BYTES + stringBytes.length);
valueBuffer.putShort((short) stringBytes.length);
valueBuffer = ByteBuffer.allocate(stringBytes.length);
valueBuffer.put(stringBytes);
break;
case BooleanArray:
keyBuffer.put((byte) 2);
boolean[] booleanArray = value.getBooleanArray();
valueBuffer = ByteBuffer.allocate(Short.BYTES + booleanArray.length);
valueBuffer.putShort((short) booleanArray.length);
valueBuffer = ByteBuffer.allocate(booleanArray.length);
for (boolean i : booleanArray) {
valueBuffer.put(i ? (byte) 1 : (byte) 0);
}
break;
case IntegerArray: // Save as IntegerArray (int32[])
keyBuffer.put((byte) 4);
case IntegerArray:
long[] intArray = value.getIntegerArray();
valueBuffer = ByteBuffer.allocate(Short.BYTES + (intArray.length * Integer.BYTES));
valueBuffer.putShort((short) intArray.length);
valueBuffer = ByteBuffer.allocate(intArray.length * Long.BYTES);
for (long i : intArray) {
valueBuffer.putInt((int) i);
valueBuffer.putLong(i);
}
break;
case FloatArray: // Save as DoubleArray
keyBuffer.put((byte) 6);
case FloatArray:
float[] floatArray = value.getFloatArray();
valueBuffer = ByteBuffer.allocate(Short.BYTES + (floatArray.length * Double.BYTES));
valueBuffer.putShort((short) floatArray.length);
valueBuffer = ByteBuffer.allocate(floatArray.length * Float.BYTES);
for (float i : floatArray) {
valueBuffer.putDouble(i);
valueBuffer.putFloat(i);
}
break;
case DoubleArray:
keyBuffer.put((byte) 6);
double[] doubleArray = value.getDoubleArray();
valueBuffer = ByteBuffer.allocate(Short.BYTES + (doubleArray.length * Double.BYTES));
valueBuffer.putShort((short) doubleArray.length);
valueBuffer = ByteBuffer.allocate(doubleArray.length * Double.BYTES);
for (double i : doubleArray) {
valueBuffer.putDouble(i);
}
break;
case StringArray:
keyBuffer.put((byte) 8);
String[] stringArray = value.getStringArray();
int capacity = Short.BYTES;
int capacity = Integer.BYTES;
for (String i : stringArray) {
capacity += Short.BYTES + i.getBytes("UTF-8").length;
capacity += Integer.BYTES + i.getBytes("UTF-8").length;
}
valueBuffer = ByteBuffer.allocate(capacity);
valueBuffer.putShort((short) stringArray.length);
valueBuffer.putInt(stringArray.length);
for (String i : stringArray) {
byte[] bytes = i.getBytes("UTF-8");
valueBuffer.putShort((short) bytes.length);
valueBuffer.putInt(bytes.length);
valueBuffer.put(bytes);
}
break;
default:
valueBuffer = ByteBuffer.allocate(0);
}

keyBuffer.putShort((short) valueBuffer.capacity());
return ByteBuffer.allocate(keyBuffer.capacity() + valueBuffer.capacity()).put(keyBuffer.array())
.put(valueBuffer.array());
} catch (UnsupportedEncodingException e) {
Expand Down
Loading
Loading