*(A: Set INT, B: Set INT): Set INT == set concat [[(a*b)$Integer for a in members A] for b in members B]

   Function declaration ?*? : (Set(Integer),Set(Integer)) -> Set(
      Integer) has been added to workspace.

2*3

Compiling function * with type (Set(Integer),Set(Integer)) -> Set(
      Integer) 
   Conversion failed in the compiled user function * .

   Cannot convert from type Integer to Set(Integer) for value
   2

This problem is not related to *.

mulset(A: Set INT, B: Set INT): Set INT == set concat [[(a*b)$Integer for a in members A] for b in members B]

   Function declaration mulset : (Set(Integer),Set(Integer)) -> Set(
      Integer) has been added to workspace.

mulset(2,3)

Compiling function mulset with type (Set(Integer),Set(Integer)) -> 
      Set(Integer) 
   Conversion failed in the compiled user function mulset .

   Cannot convert from type Integer to Set(Integer) for value
   2

It works if conversion from Integer to set is explicit.

set [2] * set [3]

(1)

But you have overridden the default behaviour in the interpreter. It is ok if you are explicit:

(2*3)$Integer

(2)

So I think probably the current behavior is sensible.