ConvertTo-Json doesn't serialize simple objects properly
Given a simple array which has been boxed in a PSObject
[PSObject]$o = @(@(Foo,1),@(Bar,2))
Calling ConvertTo-Json returns a weird pointless wrapper around it:
convertto-json $o
{
"value": [
[
"Foo",
1
],
[
"Bar",
2
]
],
"Count": 2
}
Which completely breaks round-trip support
ConvertFrom-Json (Convertto-Json $o)
value Count
{Foo 1, Bar 2} 2
Joel "Jaykul" Bennett
shared this idea
14 comments
-
Paal Braathen
commented
I might agree, but I find the behavior confusing.
Does anyone know what actually happens with an object when you "cast" it to [PSObject]?
These objects seem equal, but somehow aren't:
PS M:\> [PSObject] $o = @(@("Foo",1),@("Bar",2))
PS M:\> [System.Array] $a = @(@("Foo",1),@("Bar",2))
PS M:\> $o.GetType()
IsPublic IsSerial Name BaseType
-------- -------- ---- --------
True True Object[] System.ArrayPS M:\> $a.GetType()
IsPublic IsSerial Name BaseType
-------- -------- ---- --------
True True Object[] System.ArrayPS M:\> ConvertTo-Json -Compress $o
{"value":[["Foo",1],["Bar",2]],"Count":2}PS M:\> ConvertTo-Json -Compress $a
[["Foo",1],["Bar",2]] -
PetSerAl
commented
@Joel "Jaykul" Bennett
Can I ask one last question? Is anything "wrong" left after this command?Remove-TypeData System.Array
-
Kirk Munro
commented
IMHO ConvertTo-Json/ConvertFrom-Json functions a little like taking a sentence or paragraph through an online translator to and from different languages over and over. I just took part of Joel's example and did the following:
PS C:\Users\Poshoholic\Documents> rv x
PS C:\Users\Poshoholic\Documents> '[
>> [
>> "Foo",
>> 1
>> ],
>> [
>> "Bar",
>> 2
>> ]
>> ]' | ConvertFrom-Json -OutVariable x > $null
PS C:\Users\Poshoholic\Documents> $x.GetType().FullName
System.Collections.ArrayList
PS C:\Users\Poshoholic\Documents> ,$x | gm CountTypeName: System.Collections.ArrayList
Name MemberType Definition
---- ---------- ----------
Count Property int Count {get;}PS C:\Users\Poshoholic\Documents> ConvertTo-Json $x
[
{
"value": [
[
"Foo",
1
],
[
"Bar",
2
]
],
"Count": 2
}
]This makes no sense. If the original JSON I gave ConvertFrom-Json converts to an ArrayList, which does not have any extended members added to it, then why does ConvertTo-Json give me JSON that now represents an object array with one member, a PSCustomObject, with a value object array (that in turn contains two object arrays) and a count property. Wtf?
-
Joel "Jaykul" Bennett
commented
Wrong: serializing an array as an object with a value property that contains the correct json representation of the array.
Wrong: serializing built-in properties of arrays (like "Length" or "IsReadOnly" or "Count") at all. The cmdlet correctly ignores the built-in "Length" property, but not the built in "Count" alias property which PowerShell adds.
Wrong: being unable to round-trip simple arrays of strings and numbers
Anyway. I'm done arguing this. I don't think even the original developer would have defended it as long as you have.
It can't be used to serialize simple arrays provided by users to pass them to JSON APIs, because you can't count on them being serialized the way JSON requires arrays to be serialized, so I had to write my own ConvertTo-Json for my customers. That's frustrating, but I guess it's a selling point for my work...
-
PetSerAl
commented
>Why are you still arguing that it is not wrong?
Define "wrong". For me "wrong" is when function does not do what it is documented to do. Does ConvertTo-Json documented to behave as CliXML serializer? Does it produce not well formed JSON? Does resulting JSON is not related with object being serialized?There are more than one choice how to serialize object to JSON, and always will be someone who expect that different choice will be chosen. Currently, I only see one "wrong" thing: ConvertTo-Json is not document choices which it made.
-
PetSerAl
commented
@Joel "Jaykul" Bennett
I do not post any workaround, I only demonstrating that extended properties are not included in JSON if object is not wrapped in PSObject. And I do not think that behavior of ConvertTo-Json is wrong. BTW, do you have any problems besides Count extended property of the System.Array type? If it is your only problem, then my workaround would be `Remove-TypeData System.Array`. That will remove Count extended property (which is unnecessary since v3) from arrays and make them produce JSON which you wish even when wrapped in PSObject. -
Joel "Jaykul" Bennett
commented
PowerShell wraps things in PSObject whenever it chooses, and when dealing with objects that have been passed as parameters or on the pipeline, one cannot simply use the BaseObject -- care must be taken to do the right thing.
The CliXML serializer ***does the right thing***
The JSON serializer does NOT. You can see this is the case by simply testing the workaround I posted earlier (serialize to CliXml, then deserialize, and then convertto-json).
Why are you still arguing that it is not wrong?
@PetSerAl Your workaround in this last post is completely incorrect. First: it will only work to unwrap the first level of an object. Second: if the thing is a PSCustomObject (including anything returned by a remote session, or imported from CliXML, etc) you would throw away ALL of the information. You most certainly would NOT be able to access any of it's properties.
-
PetSerAl
commented
And ConvertTo-Json cares about extended properties only when it sees PSObject. So, it is simple rule: do not wrap your objects into PSObject if you do not want to see extended properties in JSON.
$MyListUnwrapped=$MyList.PSObject.BaseObject
# you still can use extended properties
$MyListUnwrapped.ExtendedProperty
# but them not in JSON now
ConvertTo-Json $MyListUnwrapped -
PetSerAl
commented
@Roman Kuzmin
If object have know JSON representation (IEnumerable, IDictionary, String, etc.), then ConvertTo-Json cares only about extended properties:Add-Type -TypeDefinition @'
public class MyList : System.Collections.Generic.List<int> {
public string NonExtendedProperty { get; set; }
}
'@$MyList=New-Object MyList|Add-Member ExtendedProperty ExtendedValue -PassThru
$MyList.NonExtendedProperty='NonExtendedValue'$MyList.AddRange([int[]](1..3))
ConvertTo-Json $MyList
And Count is extended property for arrays in PowerShell.
-
Joel "Jaykul" Bennett
commented
ConvertTo-Json should be as smart about PSObjects as ConvertTo-CliXml is.
So essentially, my proposal is that ConvertTo-Json should work the way it does if you first pipe things through CliXml.
For instance, compare these objects:
[PsObject]$po = @("Foo",2)
[PSObject[]]$po2 = @("Foo",2),@("Bar",4)
Compare:
ConvertTo-Json $po
ConvertTo-Json $($po)
ConvertTo-Json $po2
ConvertTo-Json $($po2)Now do the same thing but first serialize them to clixml:
function cvjson($io) {
$TP = [IO.Path]::GetTempFileName()
Export-CliXml -InputObject $io -LiteralPath $TP
Import-CliXml -LiteralPath $TP | ConvertTo-json
Remove-Item $TP
}cvjson $po
cvjson $($po)
cvjson $po2
cvjson $($po2)You will see that abusing the CliXml serializer properly unwraps PSObjects when serializing them, producing the correct expected output every time. P.S. I don't know why wrapping the PSObject in $() fixes the bug at one level, but it doesn't work recursively, only using the CliXml serialize does.
-
Roman Kuzmin
commented
I believe the pair ConvertTo-Json and ConvertFrom-Json is supposed to serialize
and deserialize data, and the data should be semantically compatible, at least
to some extent (JSON is not that flexible).The current effort of ConvertTo-Json to convert custom properties of standard
types has low value. Rehydrated values are changed completely. The code should
know when it deals with the original data and when with rehydrated.To deal with this, a code writer should redesign the objects, e.g. create
proper property bags, aggregate a standard object (e.g. a collection), add
other custom properties. In other words, to avoid what ConvertFrom-Json is
doing. So why is this needed at all?All in all, I vote for simple and clear ConvertTo-Json which converts just base
objects without extra properties unless the base object is PSCustomObject, i.e.
a PSObject is a true custom object. If users need to store and restore extra
properties they use custom objects. Just like now, if rehydrated objects should
be similar to the original (which is probably the case). -
Roman Kuzmin
commented
So one possible and easy to document way is: ConvertTo-Json does not convert
custom properties unless they are properties of a true property bag, i.e. a
PSObject wrapping PSCustomObject. It is not perfect but it is very clear.
And it is very simple to implement.On the other hand, if ConvertTo-Json should support any objects with custom
properties then it is still a lot of work to do. And this would be an effort
for fragile and rather rare cases of adding custom properties to standard
types. The current approach has current and future issues (see my examples). I
do not see, for example how to easily resolve conflicts with potential user
properties named 'value' or 'Value'. It is not a rare name, by the way. -
Roman Kuzmin
commented
@PetSerAl,
Good question.Your example shows the dilemma, indeed. Even though adding PS custom properties
to standard types is a fragile approach, IMHO. If I am not mistaken, there
were reported scenarios when such properties got lost on various operations.
ConvertTo-Json may loose them too right now, see examples.The behaviour should be at least documented and consistent across types.
Some contrived examples with inconsistencies and issues:
------8<-------
# ExtraPropertyValue, value, Count : why Count? value already represents a wrapped object
ConvertTo-Json (,(1,2) | Add-Member ExtraPropertyName ExtraPropertyValue -PassThru)# ExtraPropertyValue, value : note that there is no Length (like Count in the above)
ConvertTo-Json ('zz' | Add-Member ExtraPropertyName ExtraPropertyValue -PassThru)# ExtraPropertyValue, a, b : why not wrapped by value?
ConvertTo-Json (@{a=1; b=1} | Add-Member ExtraPropertyName ExtraPropertyValue -PassThru)# a, b : not wrapped and a=42 lost (not a surprise but it would not be lost if value is used)
ConvertTo-Json (@{a=1; b=1} | Add-Member a 42 -PassThru)# value, Count : note that the custom value=42 is lost
ConvertTo-Json (,(1,2) | Add-Member value 42 -PassThru)# value, Value, Count : this JSON cannot be imported by ConvertFrom-Json
$json = ConvertTo-Json (,(1,2) | Add-Member Value 42 -PassThru)
$json | ConvertFrom-Json------8<-------
-
PetSerAl
commented
So, what is your proposal? Should extra properties on collection types just be ignored? Or should them be converted to JSON somehow else? Or should <code>Count</code> array alias property to be special exception? What JSON following command should produce?
ConvertTo-Json (,(1..3)|Add-Member ExtraPropertyName ExtraPropertyValue -PassThru)