byteArrayToByteString, and byteStringToByteArray?

[chessai]

i find myself using the following more often now:

unsafeByteArrayToByteString :: ByteArray -> ByteString
unsafeByteArrayToByteString (ByteArray b#) =
  let addr# = byteArrayContents# b#
       fp = ForeignPtr addr# (PlainPtr (unsafeCoerce# b#))
       len = I# (sizeofByteArray# b#)
  in PS fp 0 len

byteArrayToByteString :: ByteArray -> ByteString
byteArrayToByteString b@(ByteArray b#)
  | isTrue# (isByteArrayPinned# b#) = unsafeByteArrayToByteString b
  | otherwise = runST $ do
      let len = sizeofByteArray b
      marr@(MutableByteArray marr#) <- newPinnedByteArray len
      copyByteArray marr 0 b 0 len
      let addr# = byteArrayContents# (unsafeCoerce# marr#)
      let fp = ForeignPtr addr# (PlainPtr (unsafeCoerce# marr#))
      pure (PS fp 0 len)

byteStringToByteArray :: ByteString -> ByteArray
byteStringToByteArray (PS fp off len) =
  accursedUnutterablePerformIO $
    withForeignPtr fp $ \ptr -> do
      marr <- newByteArray len
      copyPtrToMutableByteArray (ptr `plusPtr` off) marr 0 len
      unsafeFreezeByteArray marr

is it possible that these could be added somewhere? i work with ByteArray a lot more than i do bytestring, but there are places where ByteString is necessary because of other apis (aeson, binary, etc.)

[hvr]

Isn't this basically the conversion to/from ShortByteString? If I understand it correctly, ByteArray is the same as

data ShortByteString = SBS ByteArray#

(slightly related, in the past I also used http://hackage.haskell.org/package/bytestring-plain-0.1.0.2/docs/Data-ByteString-Plain.html before there was ShortByteString which had a different cost-model, but since ShortByteString I almost always use that one -- probably in the very use-cases you seem to prefer ByteArray )

[Bodigrim]

I think @hvr is right, one can employ fromShort / toShort.

import Data.ByteString.Short.Internal (ShortByteString(..), fromShort, toShort)

byteArrayToByteString :: ByteArray -> ByteString
byteArrayToByteString (ByteArray b#) = fromShort (SBS b#)

byteStringToByteArray :: ByteString -> ByteArray
byteStringToByteArray bs = let SBS b# = toShort bs in ByteArray b#

[chessai]

Revisiting this, I believe it is possible to do this without copying, if the ForeignPtr allows it. Some example code:

{-# language MagicHash, UnboxedTuples #-}

import Data.ByteString.Internal (ByteString(..))
import Data.ByteString.Short (ShortByteString(..), toShort)
import GHC.ForeignPtr (ForeignPtr(..), ForeignPtrContents(..))
import GHC.Exts
  ( MutableByteArray#, ByteArray#, Addr#, Int(..), RealWorld,
    mutableByteArrayContents#, eqAddr#, isTrue#, runRW#, unsafeFreezeByteArray#,
    getSizeofMutableByteArray#, (==#)
  )
import Data.Primitive.ByteArray(ByteArray(..))

import qualified Data.ByteString as BS
import qualified Data.List as List
import qualified Data.Char as Char
import Data.Word (Word8)
import GHC.Exts (toList)

byteStringToByteArray :: ByteString -> ByteArray
byteStringToByteArray b@(BS (ForeignPtr addr# contents) (I# len#)) = case contents of
  MallocPtr marr# _ -> ByteArray (go marr# addr#)
  PlainPtr marr# -> ByteArray (go marr# addr#)
  _ -> ByteArray (byteStringToByteArrayCopy b)

  where
    go :: MutableByteArray# RealWorld -> Addr# -> ByteArray#
    go marr# addr# =
      let marrAddr# = mutableByteArrayContents# marr#
      in if isTrue# (eqAddr# addr# marrAddr#)
         then runRW#
                (\s0 -> case getSizeofMutableByteArray# marr# s0 of
                  (# s1, marrLen# #) ->
                    if isTrue# (marrLen# ==# len#)
                    then case unsafeFreezeByteArray# marr# s1 of
                           (# _, r #) -> r
                    else byteStringToByteArrayCopy b
                )
         else byteStringToByteArrayCopy b

    byteStringToByteArrayCopy :: ByteString -> ByteArray#
    byteStringToByteArrayCopy s = case toShort s of { SBS arr -> arr; }

main :: IO ()
main = do
  let b = BS.pack [0x20..0x7e]
  let r = byteStringToByteArray b
  
  let byteToChar byte = Char.chr (fromIntegral @Word8 @Int byte)
  let bChars = List.map byteToChar (BS.unpack b)
  let rChars = List.map byteToChar (toList r)
  
  print (bChars == rChars)

This can just return a ShortByteString instead if ByteString doesn't want to concern itself with lifted ByteArray.

[clyring]

#547 does some similar stuff specifically to better support IO operations. As discussed there, I have serious reservations about the idea because of its surprising interaction with compact regions.

But there are several tricky things about it to get right about the implementation, and I suppose it is a rather fundamental operation. Perhaps a version of it can live in Data.ByteString.Unsafe? Feel free to open a pull request.

[phadej]

I found myself writing today the following, note: this is converting MutableByteArray.

-- | See implementation of 'GHC.Foreign.Ptr.mallocForeignPtrBytes'
--
-- MutableByteArray should not be modified after creation of the ByteString
pinnedMutableByteArrayToBS :: MutableByteArray RealWorld -> Int -> ByteString
pinnedMutableByteArrayToBS (MutableByteArray ba) len = 
    BS (ForeignPtr (mutableByteArrayContents# ba) (PlainPtr ba)) len

Arguably, there could be MutableByteArray RealWorld -> ForeignPtr in base (or ghc-experimental); (AFAICT that is "safe" function, just providing different interface to mutable memory region), but I won't push for that.

So the unsafety of pinnedMutableByteArrayToBS is really related to using BS constructor.

I don't want to go via ShortByteString route, as I know that my bytearray is pinned.

There's also https://gitlab.haskell.org/ghc/ghc/-/issues/22255 which if resolved would mean that ShortByteString would not need to check for pinnedness, as ShortByteString are never pinned (i.e. "Compaction considers a ByteArray# to be pinned only if it was explicitly pinned at creation-time")