• nef@slrpnk.net
    link
    fedilink
    English
    arrow-up
    9
    ·
    1 day ago

    This looks mostly right. The precision slider is definitely only for the output, not calculations. The (up - 5/4 - down) is (always round up - round anything less than 0.5 down - always round down)

    What I’d like to know is how the A and F settings are different.

    • Benjaben@lemmy.world
      link
      fedilink
      arrow-up
      3
      ·
      23 hours ago

      Auto to me (if A is Auto) sounds like it’d truncate unnecessary digits (4 or 4.0 instead of 4.0000) maybe? Whereas if F is Full then you’d get full precision?

      Idk seems logical but not especially useful, probably not a great guess.

      • no banana@lemmy.worldOP
        link
        fedilink
        arrow-up
        1
        ·
        6 hours ago

        I can say that A seems to be Auto in the way you think but I haven’t figured out what criteria it has for this automation. Maybe there’s a way I could figure it out. I’m not a maths kind of guy (had pretty bad teachers in school) so it’s all honestly above my head. Shamefully enough.

      • lime!@feddit.nu
        link
        fedilink
        English
        arrow-up
        2
        ·
        23 hours ago

        i’ve never seen one of these display decimals like that, it doesn’t seem like something that would need a setting.