Please feel free to provide feedback or file bugs here.

ConvertTo-Json doesn't serialize simple objects properly

# Given a simple array which has been boxed in a PSObject
[PSObject]$o = @(@(Foo,1),@(Bar,2))

# Calling ConvertTo-Json returns a weird pointless wrapper around it:
convertto-json $o
{
"value": [
[
"Foo",
1
],
[
"Bar",
2
]
],
"Count": 2
}

# Which completely breaks round-trip support
ConvertFrom-Json (Convertto-Json $o)

value Count
----- -----
{Foo 1, Bar 2} 2

16 votes
Sign in
(thinking…)
Password icon
Signed in as (Sign out)

We’ll send you updates on this idea

Joel "Jaykul" Bennett shared this idea  ·   ·  Flag idea as inappropriate…  ·  Admin →

14 comments

Sign in
(thinking…)
Password icon
Signed in as (Sign out)
Submitting...
  • Paal Braathen commented  ·   ·  Flag as inappropriate

    I might agree, but I find the behavior confusing.

    Does anyone know what actually happens with an object when you "cast" it to [PSObject]?

    These objects seem equal, but somehow aren't:

    PS M:\> [PSObject] $o = @(@("Foo",1),@("Bar",2))

    PS M:\> [System.Array] $a = @(@("Foo",1),@("Bar",2))

    PS M:\> $o.GetType()

    IsPublic IsSerial Name BaseType
    -------- -------- ---- --------
    True True Object[] System.Array

    PS M:\> $a.GetType()

    IsPublic IsSerial Name BaseType
    -------- -------- ---- --------
    True True Object[] System.Array

    PS M:\> ConvertTo-Json -Compress $o
    {"value":[["Foo",1],["Bar",2]],"Count":2}

    PS M:\> ConvertTo-Json -Compress $a
    [["Foo",1],["Bar",2]]

  • PetSerAl commented  ·   ·  Flag as inappropriate

    @Joel "Jaykul" Bennett
    Can I ask one last question? Is anything "wrong" left after this command?

    Remove-TypeData System.Array

  • Kirk Munro commented  ·   ·  Flag as inappropriate

    IMHO ConvertTo-Json/ConvertFrom-Json functions a little like taking a sentence or paragraph through an online translator to and from different languages over and over. I just took part of Joel's example and did the following:

    PS C:\Users\Poshoholic\Documents> rv x
    PS C:\Users\Poshoholic\Documents> '[
    >> [
    >> "Foo",
    >> 1
    >> ],
    >> [
    >> "Bar",
    >> 2
    >> ]
    >> ]' | ConvertFrom-Json -OutVariable x > $null
    PS C:\Users\Poshoholic\Documents> $x.GetType().FullName
    System.Collections.ArrayList
    PS C:\Users\Poshoholic\Documents> ,$x | gm Count

    TypeName: System.Collections.ArrayList

    Name MemberType Definition
    ---- ---------- ----------
    Count Property int Count {get;}

    PS C:\Users\Poshoholic\Documents> ConvertTo-Json $x
    [
    {
    "value": [
    [
    "Foo",
    1
    ],
    [
    "Bar",
    2
    ]
    ],
    "Count": 2
    }
    ]

    This makes no sense. If the original JSON I gave ConvertFrom-Json converts to an ArrayList, which does not have any extended members added to it, then why does ConvertTo-Json give me JSON that now represents an object array with one member, a PSCustomObject, with a value object array (that in turn contains two object arrays) and a count property. Wtf?

  • Joel "Jaykul" Bennett commented  ·   ·  Flag as inappropriate

    Wrong: serializing an array as an object with a value property that contains the correct json representation of the array.

    Wrong: serializing built-in properties of arrays (like "Length" or "IsReadOnly" or "Count") at all. The cmdlet correctly ignores the built-in "Length" property, but not the built in "Count" alias property which PowerShell adds.

    Wrong: being unable to round-trip simple arrays of strings and numbers

    Anyway. I'm done arguing this. I don't think even the original developer would have defended it as long as you have.

    It can't be used to serialize simple arrays provided by users to pass them to JSON APIs, because you can't count on them being serialized the way JSON requires arrays to be serialized, so I had to write my own ConvertTo-Json for my customers. That's frustrating, but I guess it's a selling point for my work...

  • PetSerAl commented  ·   ·  Flag as inappropriate

    >Why are you still arguing that it is not wrong?
    Define "wrong". For me "wrong" is when function does not do what it is documented to do. Does ConvertTo-Json documented to behave as CliXML serializer? Does it produce not well formed JSON? Does resulting JSON is not related with object being serialized?

    There are more than one choice how to serialize object to JSON, and always will be someone who expect that different choice will be chosen. Currently, I only see one "wrong" thing: ConvertTo-Json is not document choices which it made.

  • PetSerAl commented  ·   ·  Flag as inappropriate

    @Joel "Jaykul" Bennett
    I do not post any workaround, I only demonstrating that extended properties are not included in JSON if object is not wrapped in PSObject. And I do not think that behavior of ConvertTo-Json is wrong. BTW, do you have any problems besides Count extended property of the System.Array type? If it is your only problem, then my workaround would be `Remove-TypeData System.Array`. That will remove Count extended property (which is unnecessary since v3) from arrays and make them produce JSON which you wish even when wrapped in PSObject.

  • Joel "Jaykul" Bennett commented  ·   ·  Flag as inappropriate

    PowerShell wraps things in PSObject whenever it chooses, and when dealing with objects that have been passed as parameters or on the pipeline, one cannot simply use the BaseObject -- care must be taken to do the right thing.

    The CliXML serializer ***does the right thing***

    The JSON serializer does NOT. You can see this is the case by simply testing the workaround I posted earlier (serialize to CliXml, then deserialize, and then convertto-json).

    Why are you still arguing that it is not wrong?

    @PetSerAl Your workaround in this last post is completely incorrect. First: it will only work to unwrap the first level of an object. Second: if the thing is a PSCustomObject (including anything returned by a remote session, or imported from CliXML, etc) you would throw away ALL of the information. You most certainly would NOT be able to access any of it's properties.

  • PetSerAl commented  ·   ·  Flag as inappropriate

    And ConvertTo-Json cares about extended properties only when it sees PSObject. So, it is simple rule: do not wrap your objects into PSObject if you do not want to see extended properties in JSON.

    $MyListUnwrapped=$MyList.PSObject.BaseObject
    # you still can use extended properties
    $MyListUnwrapped.ExtendedProperty
    # but them not in JSON now
    ConvertTo-Json $MyListUnwrapped

  • PetSerAl commented  ·   ·  Flag as inappropriate

    @Roman Kuzmin
    If object have know JSON representation (IEnumerable, IDictionary, String, etc.), then ConvertTo-Json cares only about extended properties:

    Add-Type -TypeDefinition @'
    public class MyList : System.Collections.Generic.List<int> {
    public string NonExtendedProperty { get; set; }
    }
    '@

    $MyList=New-Object MyList|Add-Member ExtendedProperty ExtendedValue -PassThru
    $MyList.NonExtendedProperty='NonExtendedValue'

    $MyList.AddRange([int[]](1..3))

    ConvertTo-Json $MyList

    And Count is extended property for arrays in PowerShell.

  • Joel "Jaykul" Bennett commented  ·   ·  Flag as inappropriate

    ConvertTo-Json should be as smart about PSObjects as ConvertTo-CliXml is.

    So essentially, my proposal is that ConvertTo-Json should work the way it does if you first pipe things through CliXml.

    For instance, compare these objects:

    [PsObject]$po = @("Foo",2)

    [PSObject[]]$po2 = @("Foo",2),@("Bar",4)

    Compare:

    ConvertTo-Json $po
    ConvertTo-Json $($po)
    ConvertTo-Json $po2
    ConvertTo-Json $($po2)

    Now do the same thing but first serialize them to clixml:

    function cvjson($io) {
    $TP = [IO.Path]::GetTempFileName()
    Export-CliXml -InputObject $io -LiteralPath $TP
    Import-CliXml -LiteralPath $TP | ConvertTo-json
    Remove-Item $TP
    }

    cvjson $po
    cvjson $($po)
    cvjson $po2
    cvjson $($po2)

    You will see that abusing the CliXml serializer properly unwraps PSObjects when serializing them, producing the correct expected output every time. P.S. I don't know why wrapping the PSObject in $() fixes the bug at one level, but it doesn't work recursively, only using the CliXml serialize does.

  • Roman Kuzmin commented  ·   ·  Flag as inappropriate

    I believe the pair ConvertTo-Json and ConvertFrom-Json is supposed to serialize
    and deserialize data, and the data should be semantically compatible, at least
    to some extent (JSON is not that flexible).

    The current effort of ConvertTo-Json to convert custom properties of standard
    types has low value. Rehydrated values are changed completely. The code should
    know when it deals with the original data and when with rehydrated.

    To deal with this, a code writer should redesign the objects, e.g. create
    proper property bags, aggregate a standard object (e.g. a collection), add
    other custom properties. In other words, to avoid what ConvertFrom-Json is
    doing. So why is this needed at all?

    All in all, I vote for simple and clear ConvertTo-Json which converts just base
    objects without extra properties unless the base object is PSCustomObject, i.e.
    a PSObject is a true custom object. If users need to store and restore extra
    properties they use custom objects. Just like now, if rehydrated objects should
    be similar to the original (which is probably the case).

  • Roman Kuzmin commented  ·   ·  Flag as inappropriate

    So one possible and easy to document way is: ConvertTo-Json does not convert
    custom properties unless they are properties of a true property bag, i.e. a
    PSObject wrapping PSCustomObject. It is not perfect but it is very clear.
    And it is very simple to implement.

    On the other hand, if ConvertTo-Json should support any objects with custom
    properties then it is still a lot of work to do. And this would be an effort
    for fragile and rather rare cases of adding custom properties to standard
    types. The current approach has current and future issues (see my examples). I
    do not see, for example how to easily resolve conflicts with potential user
    properties named 'value' or 'Value'. It is not a rare name, by the way.

  • Roman Kuzmin commented  ·   ·  Flag as inappropriate

    @PetSerAl,
    Good question.

    Your example shows the dilemma, indeed. Even though adding PS custom properties
    to standard types is a fragile approach, IMHO. If I am not mistaken, there
    were reported scenarios when such properties got lost on various operations.
    ConvertTo-Json may loose them too right now, see examples.

    The behaviour should be at least documented and consistent across types.

    Some contrived examples with inconsistencies and issues:

    ------8<-------

    # ExtraPropertyValue, value, Count : why Count? value already represents a wrapped object
    ConvertTo-Json (,(1,2) | Add-Member ExtraPropertyName ExtraPropertyValue -PassThru)

    # ExtraPropertyValue, value : note that there is no Length (like Count in the above)
    ConvertTo-Json ('zz' | Add-Member ExtraPropertyName ExtraPropertyValue -PassThru)

    # ExtraPropertyValue, a, b : why not wrapped by value?
    ConvertTo-Json (@{a=1; b=1} | Add-Member ExtraPropertyName ExtraPropertyValue -PassThru)

    # a, b : not wrapped and a=42 lost (not a surprise but it would not be lost if value is used)
    ConvertTo-Json (@{a=1; b=1} | Add-Member a 42 -PassThru)

    # value, Count : note that the custom value=42 is lost
    ConvertTo-Json (,(1,2) | Add-Member value 42 -PassThru)

    # value, Value, Count : this JSON cannot be imported by ConvertFrom-Json
    $json = ConvertTo-Json (,(1,2) | Add-Member Value 42 -PassThru)
    $json | ConvertFrom-Json

    ------8<-------

  • PetSerAl commented  ·   ·  Flag as inappropriate

    So, what is your proposal? Should extra properties on collection types just be ignored? Or should them be converted to JSON somehow else? Or should <code>Count</code> array alias property to be special exception? What JSON following command should produce?

    ConvertTo-Json (,(1..3)|Add-Member ExtraPropertyName ExtraPropertyValue -PassThru)

Feedback and Knowledge Base