Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Draft] C API v2.0 #5728

Draft
wants to merge 12 commits into
base: main
Choose a base branch
from
Draft

[Draft] C API v2.0 #5728

wants to merge 12 commits into from

Conversation

mzient
Copy link
Contributor

@mzient mzient commented Nov 28, 2024

Category:

Key points relevant for the review:

Tests:

  • Existing tests apply
  • New tests added
    • Python tests
    • GTests
    • Benchmark
    • Other (build test in C language)
  • N/A

Checklist

Documentation

  • Existing documentation applies
  • Documentation updated
    • Docstring
    • Doxygen
    • RST
    • Jupyter
    • Other
  • N/A

DALI team only

Requirements

  • Implements new requirements
  • Affects existing requirements
  • N/A

REQ IDs: N/A

JIRA TASK: DALI-4124

@mzient mzient force-pushed the C_API_2 branch 4 times, most recently from e2bd65a to 789df99 Compare December 4, 2024 08:44
daliExecType_t exec_type;
daliBool enable_checkpointing;
daliBool enable_memory_stats;
} daliPipelineParams_t;
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you plan adding macros to initialize this structure? It is possible to overlook and set device_id only without setting device_id_present.
Macros like:
#define DEVICE_ID_SET(pipeline_params, device_id) pipeline_params.device_id_present = 1; pipeline_params.device_id = device_id;
could minimize such subtle errors.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I considered that, yes.

};
daliDataType_t dtype;
int ndim;
} daliPipelineOutputDesc_t;
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same here, setting dtype/ndim requires setting additional field.

const cudaStream_t *stream);

/** Gets the number of pipeline outputs */
DALI_API daliResult_t daliPipelineGetOutputCount(daliPipeline_h pipeline, int *out_count);
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Starting from this functions have different documentation style. No parameters description, and usually no return type description.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Extended.

* The function ensures that a readiness event is associated with the tensor list.
* It can also get the event handle, if the output parameter pointer is not NULL.
*/
DALI_API daliResult_t daliTensorListGetOrCreateReadyEvent(
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What are possible retvals and what do they mean are they the same as in the previous call?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Well, at this point it's still a draft so I prefer not to commit to very precise error codes. When not specified otherwise, one can expect DALI_SUCCESS on success and some error code on failure. DALI_NO_DATA is specifically documented in some of the calls to denote more than one successful result.

Before we can commit to returning specific error codes. we need to fix error handling across DALI - which means nuking most of the DALI_ENFORCE/DALI_FAIL and replacing those with proper excetpions. Currently it's a mess - we mix usage errors (invalid argument or combination thereof), data errors (data-dependent index out of range, reduction over 0 extent), system errors (file not found, network errors), logic errors and more.

* @retval DALI_SUCCESS T
*
* The pointer returned in `out_shape` remains valid until the TensorList is destroyed or modified.
* If the caller is not intersted in some of the values, the pointers can be NULL.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it expected that for the call: daliResult_t retval = daliTensorListGetShape(tl, NULL, NULL, NULL) ;, the retval will be DALI_SUCCESS?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Either that or DALI_NO_DATA. If we make the fields optional, there's no point in adding an artificial failure in a pointless call.

const char *layout
);

/** Gets the "source info" metadata of a sample */
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would appreciate more descriptive header

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fair enough.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed.

mzient and others added 12 commits December 20, 2024 16:27
Signed-off-by: Michal Zientkiewicz <[email protected]>
Signed-off-by: Michał Zientkiewicz <[email protected]>
Signed-off-by: Michał Zientkiewicz <[email protected]>
Signed-off-by: Michał Zientkiewicz <[email protected]>
Signed-off-by: Michal Zientkiewicz <[email protected]>
Signed-off-by: Michal Zientkiewicz <[email protected]>
Signed-off-by: Michal Zientkiewicz <[email protected]>
Signed-off-by: Michał Zientkiewicz <[email protected]>
Signed-off-by: Michał Zientkiewicz <[email protected]>
Signed-off-by: Michał Zientkiewicz <[email protected]>
Signed-off-by: Michal Zientkiewicz <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants