[midPoint] Development of the REST API

[Radovan Semancik]

On 04/01/2016 12:05 PM, Beat Rohrer wrote:
> One of the reason why I've called it a "thin wrapper" is, because the 
> XML seems to take precedence over anything I've provided in the URI. 
> E.g. I've POSTed to http://localhost:8080/midpoint/ws/rest/users/ XML 
> data of a new user which accidentally contained the oid of an existing 
> user in the root element. I would expect the API to respond with an 
> error, but it has just updated the existing user (and responded with 
> "201 Created").

Yes, you are right. That is a bug. Feel free to file that bug in our 
jira: https://wiki.evolveum.com/display/midPoint/Creating+a+Bug+Report

> Also, you wouldn't expect to *POST* to a sub-resource /{type}/search 
> in order to GET the data…

Actually, this is something that we might need to support. E.g. the URL 
may have size limits and the search query may not fit into that limit. 
Having an option to use POST in addition to GET for searches might not 
be strictly RESTful, but it is practical. Also our search queries are 
tree-like data structures (currently XML-formatted). Expressing that as 
an URL parameter is not very convenient. And I do not see any harm of 
providing POST for this. The REST architectural style mandates that all 
GETs much be safe operations. But it does not mandates that POSTs must 
not be safe. So having a safe operation using POST does not violate REST 
principles.

> I'd propose the following improvements:
>
> - always respond in the requested format: if the request says "Accept: 
> application/xml", please respond in application/xml and not in HTML

Yes. That is a good suggestion.

> - don't respond with an error 500 if my request actually is the 
> problem, e.g. when referencing an non-existing role oid in a create 
> user POST

Agreed. The 500 should be only used for internal errors (bugs).

> - respond with an error code and a message

Agreed. But the problem is that a short message is often not enough to 
diagnose a problem. E.g. there might be errors inside an operation, some 
of the error might be handled, other not, part of the operation was a 
success and other parts not. As midPoint is using connectors and it 
communicates with a number of external and really unpredictable systems, 
midPoint does not always understands what exactly is going on and what's 
the problem. I cannot really imagine to have a numeric code for every 
possible error condition in midPoint. Simplistic twitter-like approach 
will be a step in a good direction and it can be used to divide errors 
into classes (roughly by existing exception and SOAP fault classes that 
we already have). It is a good start, but it is not really enough.

We have a partial error and handled error status code for individual 
operations. As far as I know there is no equivalent in HTTP for that. We 
have a rich data structure (OperationResult) that is a hierarchical and 
can describe what exactly was tried and what exactly went wrong. We are 
placing that in the SOAP faults so a SOAP client can take a full 
advantage of that. But I have no idea how to do similar thing in RESTful 
service without turning the service into RPC. Any ideas?

> Additionally to the things mentioned above:
>
> - please implement JSON

Yes, it is planned for some time already. We have a prototype code for 
that. But it was constantly moved out of the release plans and it was 
never fully integrated because midPoint subscribers and contributors 
prioritized other features. And we need to listen to those that are 
funding midPoint development.

> - I'd really like to see an abstraction layer between the internal 
> interface and the API, i.e.:
>   - xmlns references should not be part of the REST interface

QNames and namespaces are concepts that are not entirely XML-specific. 
MidPoint is using QNames internally to avoid conflicts of extensions 
coming form different authors. Namespaces are also used for data model 
versioning. The "/xml/ns" part in the URL was probably not the wisest 
choice and we will most likely change it when we get to midPoint 4.0 (we 
cannot change it now without breaking compatibility). But the concepts 
of namespaces and QNames are here to stay. Even in JSON. Because JSON 
and REST-like approaches have no solution for that. And we definitely 
need safe extensibility and versioning. So, the namespaces will stay 
there. However, we have found a way how to make the use of namespaces 
optional in most cases. Even in XML. We are gradually working on that.

>   - PATCHes should be much simpler: why do I have to reference 
> objectType, changeType, oid, and itemDelta? When I'm sending a PATCH 
> to /{type}/{oid} this is already defined and the diff should be enough

The delta is our diff. Yes, objectType and OID should not be there 
because that is already in the URI. And the changeType is implicit in 
this case. You are right, these are mostly leftover from the Java and 
SOAP operations. To keep things simple we have decided to reuse the same 
data structures and forgot to clean it up later. That can be improved.

But we need item deltas. We cannot remove these. Or is there any 
representation-agnostic way how to describe a fine-grained change inside 
the web resource? I'm not aware of any.

>   - implement search as URI attributes with GET

Already mentioned above. We can do it. But we also need to keep the POST.

> - I'd expect to receive a list of all resources of a type, when 
> GETting /{type}/

Yes, that might be something to add. But, there is already a way how to 
list all objects using search resources. Therefore this would be 
duplication of functionality and it will make the API non-minimal. But 
it seems to be common practice in REST so I'm willing to consider that.

> - please improve the REST API's documentation: in the current state, 
> there is no way to understand the XML structure just from reading the 
> doc and the referenced resources; e.g. please link the resources to 
> their respective class documentation, provide a link to the search 
> query format etc.

Again, that is something that is on our TODO list for a long time. It is 
difficult to find the time to document things properly when there is a 
constant pressure for new features. But at least we are planning to have 
swager description quite soon. And the data model is quite well 
described in the XSD files and used to generate the documentation: 
https://wiki.evolveum.com/display/midPoint/SchemaDoc

Thanks for your suggestions, they are most welcome. Maybe you could 
create an "improvement" issue in our jira and summarize the suggested 
improvements there?
Most of the suggestions can be implemented quite quickly. However, this 
will need an endorsement from a midPoint subscriber to fit it into the 
roadmap: https://wiki.evolveum.com/display/midPoint/Roadmap

-- 
Radovan Semancik
Software Architect
evolveum.com